Systems and Methods for Displaying Augmented Anatomical Features

ABSTRACT

A method, user device, and system for displaying augmented anatomical features is disclosed. The method includes detecting a target individual, displaying a visual representation of the body, and determining an anatomical profile of the target individual based on a plurality of reference markers. The method further includes displaying, on the display, a graphical representation of the inner anatomical features onto the visual representation of the body so as to assist in the identification of the inner anatomical features. In another aspect, an initial three-dimensional representation of the body is mapped and a preferred anatomical profile is determined based upon the reference markers. The initial three-dimensional representation of the body is modified to be the shape of the preferred anatomical profile and displayed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. application Ser. No.16/514,163 filed on Jul. 17, 2019, the contents of which areincorporated herein.

TECHNICAL FIELD

This disclosure relates generally to a method, system and user devicefor displaying inner anatomical features.

BACKGROUND

This section provides background information related to the presentdisclosure and is not necessarily prior art.

Augmented reality technology has the ability to alter, or augment, auser's view of the surrounding environment by overlayingcomputer-generated images onto the user's view of the real world,creating a composite view consisting of both real and virtual elements.Augmented reality offers the user an enriching experience by augmenting,via overlaid digital content, the user's perception of their environmentand their immediate surroundings. The user may augment their viewthrough various electronic devices, such as wearable technology (e.g.,headsets, glasses, smart watches, etc.), tablets, laptops, mobiledevices, or other devices. The user can use these electronic devices toaugment their perception of their environment by overlaying, forinstance, information about their surroundings, or graphical images toenhance their perception of their current environment.

Augmented reality can be used in a variety of environments by a varietyof users to educate each user about their surroundings. For example, arailyard worker can wear augmented reality glasses that allow them toview information about trains in the railyard, or a biologist may useaugmented reality to identify different species of plants surroundingthem.

Healthcare professionals, such as doctors and nurses, are in continuousneed of technological assistance in order to treat their patients.Particularly, healthcare professionals constantly need to obtain andaccumulate data on their patients in order to assess the best treatmentplan for the patient. Healthcare professionals would greatly benefitfrom using augmented reality to gather data on their patients. Whileknown augmented reality technology has been used for healthcareprofessionals to gather patient data, a continuous need for improvementremains in the pertinent art.

SUMMARY

This section provides a general summary of the disclosure, and is not acomprehensive disclosure of its full scope or all of its features.

The instant disclosure provides various methods, user devices, andsystems for displaying augmented anatomical features. One aspect of thedisclosure provides a method. The method includes detecting a targetindividual. The target individual has a body. The method furtherincludes tracking the target individual. The method further includesdisplaying, on a display, a visual representation of the body. Themethod further includes identifying a plurality of reference markers onthe visual representation of the body. The method further includesdetermining, at a processor, an anatomical profile of the targetindividual. The anatomical profile of the target individual isdetermined based on the plurality of reference markers. The anatomicalprofile includes a plurality of anatomical features. The method furtherincludes displaying, on the display, graphical representations of theanatomical features overlaid on the visual representation of the body.The graphical representations of the anatomical features are oriented onthe visual representation of the body based on the anatomical profile.

Implementations of the disclosure may also include one or more of thefollowing features. In some implementations, the plurality of referencemarkers correspond to at least one of a navel, a portion of a sternum, aportion of a hip, a portion of a collarbone, or a portion of a shoulder.

In some implementations, the anatomical profile of the target individualis further based on a plurality of data corresponding to the body.

In some implementations, the plurality of anatomical features includesat least one of organs, bones, muscles, or blood vessels.

In some implementations, the plurality of reference markers areidentified by a user interacting with the display.

In some implementations, the anatomical profile of the target individualincludes a default anatomical profile that is modified based on theplurality of reference markers.

According to another aspect of the disclosure, a user device isprovided. The user device includes a display. The user device furtherincludes data processing hardware in communication with the display. Theuser device further includes memory hardware in communication with thedata processing hardware. The memory hardware stores instructions thatwhen executed on the data processing hardware cause the data processinghardware to perform operations. According to this aspect, the operationscarry out a method. The method includes detecting a target individual.The target individual has a body. The method further includes trackingthe target individual. The method further includes displaying, on thedisplay, a visual representation of the body. The method furtherincludes identifying a plurality of reference markers on the visualrepresentation of the body. The method further includes determining, ata processor, an anatomical profile of the target individual. Theanatomical profile of the target individual is determined based on theplurality of reference markers. The anatomical profile includes aplurality of anatomical features. The method further includesdisplaying, on the display, graphical representations of the anatomicalfeatures overlaid on the visual representation of the body. Thegraphical representations of the anatomical features are oriented on thevisual representation of the body based on the anatomical profile.

This aspect may also include one or more of the following features. Insome implementations, the plurality of reference markers correspond toat least one of a navel, a portion of a sternum, a portion of a hip, aportion of a collarbone, or a portion of a shoulder.

In some implementations, the anatomical profile of the target individualis further based on a plurality of data corresponding to the body.

In some implementations, the plurality of anatomical features includesat least one of organs, bones, muscles, or blood vessels.

In some implementations, the plurality of reference markers areidentified by a user interacting with the display.

In some implementations, the anatomical profile of the target individualincludes a default anatomical profile that is modified based on theplurality of reference markers.

According to another aspect of the disclosure, a system is provided. Thesystem includes a user device. The user device includes a first display.The user device further includes data processing hardware incommunication with the first display. The user device further includesmemory hardware in communication with the data processing hardware. Thememory hardware stores instructions that when executed on the dataprocessing hardware cause the data processing hardware to performoperations. According to this aspect, the operations carry out a method.The method includes, detecting a target individual. The targetindividual has a body. The method further includes tracking the targetindividual. The method further includes displaying, on the firstdisplay, a visual representation of the body. The method furtherincludes identifying a plurality of reference markers on the visualrepresentation of the body. The method further includes determining, ata processor, an anatomical profile of the target individual. Theanatomical profile of the target individual is determined based on theplurality of reference markers. The anatomical profile includes aplurality of anatomical features. The method further includesdisplaying, on the first display, graphical representations of theanatomical features overlaid on the visual representation of the body.The graphical representations of the anatomical features are oriented onthe visual representation of the body based on the anatomical profile.The system further includes a wearable in communication with the userdevice. The wearable includes a second display configured to displaygraphical representations of the anatomical features overlaid on thevisual representation of the body. The graphical representations of theanatomical features are oriented on the visual representation of thebody based on the anatomical profile.

This aspect may also include one or more of the following features. Insome implementations, the plurality of reference markers correspond toat least one of a navel, a portion of a sternum, a portion of a hip, aportion of a collarbone, or a portion of a shoulder.

In some implementations, the anatomical profile of the target individualis further based on a plurality of data corresponding to the body.

In some implementations, the plurality of anatomical features includesat least one of organs, bones, muscles, or blood vessels.

In some implementations, the plurality of reference markers areidentified by a user interacting with the display.

In some implementations, the wearable is further configured to detectand track the target individual independently of the user device.

In some implementations, the anatomical profile of the target individualincludes a default anatomical profile that is modified based on theplurality of reference markers.

According to another aspect of the disclosure, a method is provided. Themethod includes detecting a target individual. The target individual hasa body. The method further includes displaying, on a display, a visualrepresentation of the body. The method further includes identifying aplurality of reference markers on the visual representation of the body.The method further includes determining, at a processor, an anatomicalprofile of the target individual. The anatomical profile of the targetindividual is determined based on the plurality of reference markers.The anatomical profile includes a plurality of inner anatomicalfeatures. The method further includes selecting a medical procedure. Themethod further includes determining a future state anatomical profilecorresponding to the selected medical procedure. The method furtherincludes displaying, on the display, graphical representations of thevisual representation of the body modified with the inner anatomicalfeatures based on the selected medical procedure.

According to another aspect of the disclosure, a method is provided. Themethod includes detecting a target individual. The target individual hasa body. The method further includes displaying, on a display, a visualrepresentation of the body. The method further includes identifying aplurality of reference markers on the visual representation of the body.The method further includes determining, at a processor, an anatomicalprofile of the target individual. The anatomical profile of the targetindividual is determined based on the plurality of reference markers.The anatomical profile includes a plurality of inner anatomicalfeatures. The method further includes displaying, on the display, agraphical representation of the inner anatomical features onto thevisual representation of the body so as to assist in the identificationof the inner anatomical features.

Implementations of the disclosure may also include one or more of thefollowing features. In some implementations, the plurality of referencemarkers correspond to at least one of a navel, a portion of a sternum, aportion of a hip, a portion of a collarbone, a portion of a shoulder,nose, corner of the eyes, tips of the ears, chin and the like.

In some implementations, the anatomical profile of the target individualis further based on a plurality of data corresponding to the body.

In some implementations, the plurality of anatomical features includesat least one of organs, bones or muscles.

In some implementations, the plurality of reference markers areidentified by a user interacting with the display.

In some implementations, the method further includes the step ofselecting a medical procedure and determining a future state anatomicalprofile corresponding to the selected medical procedure, wherein thegraphical representation includes a visual representation of the bodymodified with the inner anatomical features based on the selectedmedical procedure. In such an implementation, the graphicalrepresentation of the body may be modified with outer anatomicalfeatures based on the selected medical procedure. In such animplementation, the method further includes providing a graphicalrepresentation of the body modified with outer anatomical features basedon the selected medical procedure. In such an implementation, theselected medical procedure is a procedure effecting at least one oforgans, bones or muscles. In such an implementation, the plurality ofreference markers correspond to a body structure such as a navel, aportion of a sternum, a portion of a hip, a portion of a collarbone, aportion of a shoulder, lips, corners of the mouth, tip of the nose, orears. In such an implementation, the anatomical profile of the targetindividual may be further based on a plurality of data corresponding tothe body. In such an implementation, the plurality of reference markersmay be identified by a user interacting with the display.

In some implementation, the method may include the step of identifyingat least one of a disease, an inherited condition or an anatomicalvariant and determining an anatomical profile corresponding to theplurality of reference markers and at least one of the identifieddisease, the identified inherited condition or the identified anatomicalvariant, wherein the graphical representation includes a visualrepresentation of the body modified with the inner anatomical featuresbased on the determined anatomical profile. In such an implementation,the disease, the inherited condition and the anatomical variant effectsat least one of organs, bones or muscles. In such an implementation, theplurality of reference markers may correspond to at least one of anavel, a portion of a sternum, a portion of a hip, a portion of acollarbone, a portion of a shoulder, lips, corners of the mouth, tip ofthe nose, or ears. In such an implementation, the anatomical profile ofthe target individual may be further based on a plurality of datacorresponding to the body. In such an implementation, the plurality ofinner anatomical features may include at least one of organs, bones ormuscles. In such an implementation, the plurality of reference markersmay be identified by a user interacting with the display.

According to another aspect of the disclosure, a user device isprovided. The user device includes a display. The user device furtherincludes data processing hardware in communication with the display. Theuser device further includes memory hardware in communication with thedata processing hardware. The memory hardware stores instructions thatwhen executed on the data processing hardware cause the data processinghardware to perform operations. According to this aspect, the operationscarry out a method. The method includes detecting a target individual.The target individual has a body. The method further includesdisplaying, on the display, a visual representation of the body. Themethod further includes identifying a plurality of reference markers onthe visual representation of the body. The method further includesdetermining, at a processor, an anatomical profile of the targetindividual. The anatomical profile of the target individual isdetermined based on the plurality of reference markers. The anatomicalprofile includes a plurality of inner anatomical features. The methodfurther includes displaying, on the display, a graphical representationof the inner anatomical features onto the visual representation of thebody so as to assist in the identification of the inner anatomicalfeatures.

This aspect may also include one or more of the following features. Insome implementations, the plurality of reference markers correspond toat least one of a navel, a portion of a sternum, a portion of a hip, aportion of a collarbone, or a portion of a shoulder, lips, corners ofthe mouth, tip of the nose, or ears.

In some implementations, the anatomical profile of the target individualis further based on a plurality of data corresponding to the body.

In some implementations, the plurality of anatomical features includesat least one of organs, bones or muscles.

In some implementations, the plurality of reference markers areidentified by a user interacting with the display.

In some implementations, the plurality of reference markers areidentified by the data processing hardware.

In some implementations, the anatomical profile of the target individualincludes a default anatomical profile that is modified based on theplurality of reference markers.

In some implementations, the operations further include selecting amedical procedure and determining a future state anatomical profilecorresponding to the selected medical procedure, wherein the graphicalrepresentation includes a visual representation of the body modifiedwith the inner anatomical features based on the selected medicalprocedure. In such an implementation, the selected medical procedure isa procedure at least one of organs, bones or muscles.

In some implementations, the operations further identify at least one ofa disease, an inherited condition or an anatomical variant anddetermining an anatomical profile corresponding to the plurality ofreference markers and at least one of the identified disease, theidentified inherited condition or the identified anatomical variant,wherein the graphical representation includes a visual representation ofthe body modified with the inner anatomical features based on thedetermined anatomical profile. In such an implementation, the disease,the inherited condition and the anatomical variant effects at least oneof organs, bones or muscles. In such an implementation, the plurality ofreference markers may correspond to at least one of a navel, a portionof a sternum, a portion of a hip, a portion of a collarbone, a portionof a shoulder, lips, corners of the mouth, tip of the nose, or ears. Insuch an implementation, the anatomical profile of the target individualmay be further based on a plurality of data corresponding to the body.In such an implementation, the plurality of inner anatomical featuresmay include at least one of organs, bones or muscles. In such animplementation, the plurality of reference markers may be identified bya user interacting with the display.

According to another aspect of the disclosure, a system is provided. Thesystem includes a user device. The user device includes a first display.The user device further includes data processing hardware incommunication with the first display. The user device further includesmemory hardware in communication with the data processing hardware. Thememory hardware stores instructions that when executed on the dataprocessing hardware cause the data processing hardware to performoperations. According to this aspect, the operations carry out a method.The method includes, detecting a target individual. The targetindividual has a body. The method further includes displaying, on thefirst display, a visual representation of the body. The method furtherincludes identifying a plurality of reference markers on the visualrepresentation of the body. The method further includes determining, ata processor, an anatomical profile of the target individual. Theanatomical profile of the target individual is determined based on theplurality of reference markers. The anatomical profile includes aplurality of inner anatomical features. The system further includes awearable in communication with the user device. The wearable includes asecond display configured to display a graphical representation of theinner anatomical features onto the visual representation of the body soas to assist in the identification of the inner anatomical features.

This aspect may also include one or more of the following features. Insome implementations, the plurality of reference markers correspond toat least one of a navel, a portion of a sternum, a portion of a hip, aportion of a collarbone, or a portion of a shoulder.

In some implementations, the anatomical profile of the target individualis further based on a plurality of data corresponding to the body.

In some implementations, the plurality of anatomical includes at leastone of organs, bones or muscles.

In some implementations, the plurality of reference markers areidentified by a user interacting with the display.

In some implementations, the wearable is further configured to detectand map the target individual independently of the user device.

In some implementations, the selected medical procedure is a procedureeffecting at least one of organs, bones or muscles.

In some implementations, the method further includes selecting a medicalprocedure and determining a future state anatomical profilecorresponding to the selected medical procedure, wherein the graphicalrepresentation includes a visual representation of the body modifiedwith the inner anatomical features based on the selected medicalprocedure.

In some implementations, the method further includes identifying atleast one of a disease, an inherited condition or an anatomical variantand determining an anatomical profile corresponding to the plurality ofreference markers and at least one of the identified disease, theidentified inherited condition or the identified anatomical variant,wherein the graphical representation includes a visual representation ofthe body modified with the inner anatomical features based on thedetermined anatomical profile. In such an implementation, the disease,the inherited condition and the anatomical variant effects at least oneof organs, bones or muscles. In such an implementation, the plurality ofreference markers may correspond to at least one of a navel, a portionof a sternum, a portion of a hip, a portion of a collarbone, a portionof a shoulder, lips, corners of the mouth, tip of the nose, or ears. Insuch an implementation, the anatomical profile of the target individualmay be further based on a plurality of data corresponding to the body.In such an implementation, the plurality of inner anatomical featuresmay include at least one of organs, bones or muscles. In such animplementation, the plurality of reference markers may be identified bya user interacting with the display.

According to another aspect of the disclosure, a method is provided. Themethod includes mapping a target individual. The target individual has abody. The method further generating an initial three-dimensionalrepresentation of the body based on the mapping, the three-dimensionalrepresentation including a plurality of anatomical features of the body.The method further includes identifying a plurality of reference markerson the visual representation of the body. The method further includesdetermining, at a processor, a preferred anatomical profile of thetarget individual. The preferred anatomical profile of the targetindividual is determined based on the plurality of reference markers.The preferred anatomical profile being a preferred three-dimensionalrepresentation of the body. The method includes modifying in threedimensions, at the processor, the initial three-dimensionalrepresentation of the body so as to have a shape of the preferredanatomical profile and displaying, on the display, the modified initialthree-dimensional representation of the body.

In some implementations, the plurality of reference markers correspondto at least one of a navel, a portion of a sternum, a portion of a hip,a portion of a collarbone, a portion of a shoulder, lips, corners of themouth, tip of the nose, or ears.

Implementations of the disclosure may also include one or more of thefollowing features. In some implementations, method includes selecting amedical procedure relating to a desired body part; and replacing thedesired body part with a corresponding desired body part taken from thepreferred anatomical profile so as to generate a future anatomical imageand displaying the future anatomical image.

In some implementations, the medical procedure is one selected from thelist consisting of a liposuction, a breast enhancement, and a tummytuck.

In some implementations, the future anatomical image is displayed inthree-dimensions.

According to another aspect of the disclosure, a user device isprovided. The user device includes a display. The user device furtherincludes data processing hardware in communication with the display. Theuser device further includes memory hardware in communication with thedata processing hardware. The memory hardware stores instructions thatwhen executed on the data processing hardware cause the data processinghardware to perform operations. According to this aspect, the operationscarry out a method. The method includes mapping a target individual. Thetarget individual has a body. The method further includes generating aninitial three-dimensional representation of the body based on themapping, the initial three-dimensional representation. The methodfurther includes identifying a plurality of reference markers on theinitial three-dimensional representation of the body. The method furtherincludes determining, at a processor, a preferred anatomical profile ofthe target individual based on the plurality of reference markers, thepreferred anatomical profile being a preferred three-dimensionalrepresentation of the body and modifying in three dimensions, at theprocessor, the initial three-dimensional representation of the body soas to have a shape of the preferred anatomical profile. The methodfurther includes displaying, on the display, a graphical representationof the modified initial three-dimensional representation of the body.

In some implementations, the plurality of reference markers correspondto at least one of a navel, a portion of a sternum, a portion of a hip,a portion of a collarbone, a portion of a shoulder, lips, corners of themouth, tip of the nose, or ears.

Implementations of the disclosure may also include one or more of thefollowing features. In some implementations, method includes selecting amedical procedure relating to a desired body part; and replacing thedesired body part with a corresponding desired body part taken from thepreferred anatomical profile so as to generate a future anatomical imageand displaying the future anatomical image.

In some implementations, the medical procedure is one selected from thelist consisting of a liposuction, a breast enhancement, and a tummytuck.

In some implementations, the future anatomical image is displayed inthree-dimensions.

According to another aspect of the disclosure, a system is provided. Thesystem includes a user device. The user device includes a first display.The user device further includes data processing hardware incommunication with the first display. The user device further includesmemory hardware in communication with the data processing hardware. Thememory hardware stores instructions that when executed on the dataprocessing hardware cause the data processing hardware to performoperations. According to this aspect, the operations carry out a method.The method includes, mapping a target individual. The target individualhas a body. The method further includes generating an initialthree-dimensional representation of the body based on the mapping, theinitial three-dimensional representation. The method further includesidentifying a plurality of reference markers on the initialthree-dimensional representation of the body. The method furtherincludes determining, at a processor, a preferred anatomical profile ofthe target individual based on the plurality of reference markers, thepreferred anatomical profile being a preferred three-dimensionalrepresentation of the body. The method further includes modifying inthree dimensions, at the processor, the initial three-dimensionalrepresentation of the body so as to have a shape of the preferredanatomical profile. The system further includes a wearable incommunication with the user device. The wearable includes a seconddisplay configured to display a graphical representation of the modifiedinitial three-dimensional representation of the body.

This aspect may also include one or more of the following features. Insome implementations, the plurality of reference markers correspond toat least one of a navel, a portion of a sternum, a portion of a hip, aportion of a collarbone, or a portion of a shoulder.

In some implementations, the plurality of reference markers correspondto at least one of a navel, a portion of a sternum, a portion of a hip,a portion of a collarbone, or a portion of a shoulder.

Implementations of the disclosure may also include one or more of thefollowing features. In some implementations, method includes selecting amedical procedure relating to a desired body part; and replacing thedesired body part with a corresponding desired body part taken from thepreferred anatomical profile so as to generate a future anatomical imageand displaying the future anatomical image.

In some implementations, the medical procedure is one selected from thelist consisting of a liposuction, a breast enhancement, and a tummytuck.

In some implementations, the future anatomical image is displayed inthree-dimensions.

The details of one or more implementations of the disclosure are setforth in the accompanying drawings and the description below. Otheraspects, features, and advantages will be apparent from the descriptionand drawings, and from the claims.

DESCRIPTION OF DRAWINGS

The drawings described herein are for illustrative purposes only ofselected configurations and not all possible implementations, and arenot intended to limit the scope of the present disclosure.

FIG. 1A illustrates an exemplary augmented reality system in accordancewith the principles of the present disclosure.

FIG. 1B illustrates an exemplary use of a user device of the augmentedreality system of FIG. 1A.

FIG. 1C illustrates an exemplary use of the augmented reality system ofFIG. 1A in accordance with the principles of the present disclosure.

FIG. 1D illustrates an exemplary use of the augmented reality system ofFIG. 1A in accordance with the principles of the present disclosure.

FIG. 1E illustrates an exemplary use of the augmented reality system ofFIG. 1A in accordance with the principles of the present disclosure.

FIG. 1F illustrates an exemplary use of the augmented reality system ofFIG. 1A in accordance with the principles of the present disclosure.

FIG. 1G illustrates an exemplary use of the augmented reality system ofFIG. 1A in accordance with the principles of the present disclosure.

FIG. 1H illustrates an exemplary use of the augmented reality system ofFIG. 1A in accordance with the principles of the present disclosure.

FIG. 1I illustrates an exemplary use of the augmented reality system ofFIG. 1A in accordance with the principles of the present disclosure.

FIG. 1J illustrates an exemplary use of the augmented reality system ofFIG. 1A in accordance with the principles of the present disclosure.

FIG. 1K illustrates an exemplary use of the augmented reality system ofFIG. 1A in accordance with the principles of the present disclosure.

FIG. 2 is a block diagram illustrating an exemplary augmented realitysystem in accordance with the principles of the present disclosure.

FIG. 3A is a block diagram illustrating an exemplary method fordisplaying augmented anatomical features in accordance with theprinciples of the present disclosure.

FIG. 3B is a block diagram illustrating an exemplary method fordisplaying a modified inner anatomical features of a patient based on aselected medical procedure.

FIG. 3C is a block diagram illustrating an exemplary method fordisplaying a three-dimensional depiction of a patient having a body of apreferred anatomical profile.

FIG. 4 is a schematic view of an example electronic device executinginstructions for displaying augmented anatomical features in accordancewith the principles of the present disclosure.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

Some of the implementations of the disclosure will be described morefully with reference to the accompanying drawings. Exampleconfigurations are provided so that this disclosure will be thorough,and will fully convey the scope of the disclosure to those of ordinaryskill in the art. Specific details are set forth such as examples ofspecific components, devices, and methods, to provide a thoroughunderstanding of configurations of the present disclosure. It will beapparent to those of ordinary skill in the art that specific detailsneed not be employed, that example configurations may be embodied inmany different forms, and that the specific details and the exampleconfigurations should not be construed to limit the scope of thedisclosure.

Example implementations provide methods, user devices, and systems fordisplaying augmented anatomical features. An augmented reality device,such as an augmented reality headset or other electronic device (e.g., aphone, a tablet computing device, or other computer), may be used tooverlay computer-generated or virtual images onto a real world view.Particularly, a healthcare professional, such as a doctor or nurse, mayuse an augmented reality device to view virtual images of anatomicalfeatures of a human body overlaid on a target individual, such as apatient, when the target individual is in view of the healthcareprofessional. The augmented reality device may project the virtualimages onto a display of the augmented reality device such that thevirtual images of anatomical features approximate one or morecharacteristics (e.g., size, location, shape, etc.) of the targetindividual's actual anatomical features. For example, the augmentedreality device may project the virtual images onto a display of theaugmented reality device such that the virtual images are located overan approximated appropriate location of the target individual's actualanatomical features according to the anatomy of the target individual.The virtual images may assist a healthcare professional in moreaccurately assessing a treatment plan or otherwise treating the patientby enhancing the health care professional's visualization of thepatient's body.

In another aspect, an augmented reality device, such as a tablet or amobile device, captures an image of the patient. The augmented realitydevice may have a software application configured to identify aplurality of reference markers on the image of the patient and determinean anatomical profile of the target individual based on the plurality ofreference markers, the anatomical profile including a plurality of inneranatomical features. The software application is further configured todisplay, on the display, a graphical representation of the inneranatomical features onto the visual representation of the body so as toassist in the identification of the inner anatomical features.

In another aspect, software application includes a list of medicalprocedures to choose from. The software application may have access to adatabase populated with a plurality of future state anatomical profilescorresponding to the selected medical procedure, wherein a graphicalrepresentation of the future state anatomical profile is overlaid on theimage of the patient and modifies the inner anatomical features based onthe selected medical procedure. Accordingly, the image capture devicedisplays how the selected medical procedure affects the inner anatomicalfeatures of the patient.

In another aspect, the image capture device is configured to map thepatient so as to generate an initial three-dimensional representation ofthe body. The software application may have access to a database that ispopulated with a plurality of preferred anatomical profiles whichcorrespond to a plurality of reference markers on the initialthree-dimensional representation of the patient. The preferredanatomical profile is a profile of a person having a preferred bodytype. The software application overlays in three dimensions theanatomical features based on the selected medical procedure with thepreferred anatomical profile and displays a graphical representation ofthe preferred three-dimensional representation of the body overlaid onthe initial three-dimensional representation of the body. Accordingly,the patient can see what he or she would look like having a preferredbody type. As this body is mapped and generated in three-dimensions, thepatient can observe the preferred body type in three-dimensions.

Referring now to FIG. 1A, an exemplary augmented reality system 10,including one or more augmented reality device(s) 102, is shown. As willbe described in more detail below, a user 104 may use the augmentedreality device(s) 102 in a healthcare environment 100 to enhance theuser's view of a target individual 106. For example, the user 104 may bea doctor, the target individual 106 may be a patient, and the healthcareenvironment 100 may be a doctor's office, such that the doctor is ableto examine the patient in the doctor's office. In another example, theuser 104 may be a paramedic providing emergency treatment to a targetindividual 106 in a healthcare environment 100 of an ambulance. Whilethe user 104 is generally shown and described herein as being ahealthcare professional (e.g., a doctor, nurse, physical therapist ortrainer, paramedic, medical assistant, pharmacist, etc.), and the targetindividual 106 is generally illustrated and described herein as being ahealthcare patient, the user 104 or target individual 106 may includevarious other persons within the scope of the present disclosure. Forexample, the individual 106 may be an athlete, student, or otherindividual that has a body and is subject to examination or study byanother user 104. In this regard, the augmented reality device(s) 102may be used in a wide range of settings by a variety of users 104 toexamine a target individual 106 in a variety of environments, includingthe healthcare environment 100.

The augmented reality device 102 may include an image capture device 108and a display 110. As will be described in more detail below, duringuse, the image capture device 108 may obtain data about the healthcareenvironment 100 and, particularly, the target individual 106 located inthe healthcare environment 100, and the display 110 may display, foruser 104 to view, a composite view of the healthcare environment 100overlaid with virtual images generated by the augmented reality device102. In some implementations, the system 10 includes a first augmentedreality device 102 a and a second augmented reality device 102 b. Thefirst augmented reality device 102 a may include a smartphone, tabletcomputer, or other suitable mobile computing device, and the secondaugmented reality device 102 b may include an augmented reality headset.References herein to the augmented reality device 102 will be understoodto apply equally to the first augmented reality device 102 a and/or thesecond augmented reality device 102 b.

The first augmented reality device 102 a may include an image capturedevice 108 a (e.g., a camera) and a display 110 a (e.g., a screen).During use, the image capture device 108 a may capture images of thehealthcare environment 100 and, particularly, the target individual 106.The screen 110 a of the first augmented reality device 102 a may be usedto display a composite view of the healthcare environment 100, capturedby the camera 108 a, and overlaid with virtual images generated by thefirst augmented reality device 102 a. The first augmented reality device102 a may include a keyboard, mouse, microphone, camera 108 a, ortouchscreen for allowing user 104 to input data to the first and/orsecond augmented reality device 102 a, 102 b.

The second augmented reality device 102 b may include an image capturedevice 108 b (e.g., a camera) and a display 110 b (e.g., an eyepiecelens). During use, the image capture device 108 b may capture images ofthe healthcare environment and, particularly, the target individual 106.The display 110 b may display a composite view of the healthcareenvironment 100, captured by camera 108 b and/or the camera 108 a, andoverlaid with virtual images generated by the second augmented realitydevice 102 b. The second augmented reality device 102 b may include atrackpad 111, camera 108 b, microphone, eye tracking device, or gesturetracking device for allowing user 104 to input data to the first and/orsecond augmented reality device 102 a, 102 b. For example, the user 104may input data and otherwise interact with the second augmented realitydevice 102 b by touch via trackpad 111; spoken commands via amicrophone; eye gestures via the camera 108 b; positional tracking ofhands or other body parts via the camera 108 b; hand gesture trackingvia the camera 108 b; or positional tracking of objects such as wands,styluses, pointers, or gloves via the camera 108 b.

Though the example shown depicts augmented reality device 102 as a firstaugmented reality device 102 a or second augmented reality device 102 b,it should be noted that augmented reality device 102 may be any device(e.g., augmented reality glasses, augmented reality helmet, tablet,etc.) capable of overlaying computer-generated or virtual images onto areal word view.

With reference now to FIG. 1B, an exemplary augmented reality device isshown. Although FIG. 1B depicts augmented reality device 102 as thefirst augmented reality device 102 a displaying visual data on screen110 a, it should be noted that, similarly, second augmented realitydevice 102 b may also display visual data on eyepiece display 110 b.Augmented reality device 102 may detect the target individual 106 byusing image capture device 108. Augmented reality device 102 may thendisplay, on display 110, a visual representation 105 of a body of thetarget individual 106. In some implementations, the visualrepresentation 105 is a live (e.g., real time) image of the targetindividual 106. In other implementations, the visual representation 105is a still image (e.g., a photograph) of the target individual 106. Insome implementations, the image capture device 108 includes an infraredcamera that uses infrared laser scatter beam technology, for example, tocreate a three-dimensional visual representation 105 of the targetindividual 106.

The augmented reality device 102 may identify (e.g., assign) one or morereference markers 112 on the visual representation 105. As describedabove, in some implementations, the reference markers 112 are identifiedon a three-dimensional visual representation 105 created using infraredlaser scatter beam technology. Each reference marker 112 may correspondto a particular part of, or location on, the body of the targetindividual 106. In some implementations, the augmented reality device102 assigns the reference marker(s) 112 by detecting an input (e.g.,touch, hand gesture, etc.) from the user 104 corresponding to one ormore particular parts of the body of the target individual 106. Inparticular, the reference markers 112 may be identified by the user's104 interaction with the augmented reality device 102. For example, insome implementations the user 104 touches the screen 110 a at locationscorresponding to each reference marker 112. In other implementations,the augmented reality device 102 b receives an input from the user 104via the camera 108 b, or the trackpad 111 corresponding to eachreference marker 112. For example, the camera 108 b may capture thelocation of the user's 104 hand at locations corresponding to eachreference marker 112.

In some implementations, the augmented reality device 102 recognizes andassigns the reference marker(s) 112 to one or more particular parts ofthe body (e.g., facial features) of the target individual 106. Forexample, the image capture device 108 may include an infrared camerathat uses infrared laser scatter beam technology, for example, torecognize and assign the reference marker(s) 112 to the one or moreparticular parts of the body (e.g., facial features) of the targetindividual 106. In particular, the image capture device 108 may be ableto create a three-dimensional reference map of the face of the targetindividual 106 and compare the three-dimensional reference map toreference data stored in a storage resource of the augmented device 102,such as the storage device 430 (FIG. 4). The augmented reality device102 may use the infrared camera of the image capturing device 108 toidentify the reference markers 112 on the face of the target individual106. The augmented reality device 102 may identify the reference markers112 on the lips, corners of the mouth, tip of the nose, or ears of thetarget individual 106. For example, the augmented reality device 102 mayidentify the reference markers 112 based on input (e.g., touch, handgesture, etc.) from the user 104. As will be explained in more detailbelow, in some implementations, the augmented device 102 uses theidentification information from the infrared camera, along with theidentified referenced markers 112 based on the input from the user 104,to transmit data corresponding to the location of the reference markers112 to a processing module (e.g., processor 410 of FIG. 4) to allow theaugmented reality device 102 to advantageously give more individualizedand specific estimates of the location of various anatomical features onthe body (e.g., face) of the target individual 106, including theunderlying blood vessels, nerves, and muscles.

In some implementations, the augmented reality device 102 identifies andassigns the reference marker(s) 112 by using machine learning orartificial intelligence algorithms to identify particular parts of thebody of the target individual 106. The augmented reality device 102 mayassign the locations of the reference markers 112 on the targetindividual 106 based on the locations of similar reference markers 112on one or more other target individuals 106. The augmented realitydevice 102 may use machine learning or artificial intelligencealgorithms to identify the target individual 106 as being a human bodyby detecting a silhouette of the target individual 106, recognizing bodyparts of the detected silhouette (e.g., limbs, crotch, armpits, orneck), and then determining the location of, and assigning, referencemarkers 112 based on the recognized body parts.

In the example shown, a first reference marker 112 a corresponds to anavel of the target individual 106. A second reference marker 112 bcorresponds to a portion of a right shoulder of the target individual106. A third reference marker 112 c corresponds to a portion of a leftshoulder of the target individual 106. A fourth reference marker 112 dcorresponds to a portion of a collarbone of the target individual 106. Afifth reference marker 112 e corresponds to a portion of a left hip ofthe target individual 106. A sixth reference marker 112 f corresponds toa portion of a right hip of the target individual 106. A seventhreference marker 112 g corresponds to a portion of a sternum of thetarget individual 106. Reference markers 112 a-112 g do not represent anexhaustive list of all reference markers, but rather an exemplary listof reference markers that may be identified by augmented reality device102. Furthermore, augmented reality device 102 may identify otherreference markers in addition to reference markers 112 a-112 g, such asreference markers corresponding to a neck, a nose, eyes, a mouth, knees,ankles, a gluteal fold, shoulder blades, wrists, or elbows, withoutdeparting from the teachings herein. Augmented reality device 102 mayalso omit one or more reference markers 112 a-112 g, without departingfrom the teachings herein.

The augmented reality device 102 may determine an anatomical profile ofthe target individual 106. The anatomical profile may include aplurality of characteristics corresponding to the individual 106. Insome implementations, the anatomical profile includes or is based on aplurality of target data, such as age or sex of the target individual106. In some implementations, the augmented reality device 102determines the anatomical profile based on an input (e.g., touch, handgesture, etc.) from the user 104. In other implementations, theaugmented reality device 102 uses machine learning or artificialintelligence algorithms to determine the anatomical profile. Forexample, the augmented reality device 102 may determine the anatomicalprofile based on a plurality of target data (e.g., the plurality ofreference markers 112) received by the augmented reality device 102.

Referring now to FIG. 1C, exemplary augmented reality devices are shown.The augmented reality devices 102 a, 102 b may display a graphicalrepresentation 113 of the target individual 106. In someimplementations, the augmented reality device 102 may display thegraphical representation 113 even when the target individual 106 isfully clothed. This is advantageous as it saves time because the targetindividual 106 does not have to remove their clothing. The graphicalrepresentations 113 of the target individual may include one or morevirtual images 114 a-114 c of anatomical features. For example, thevirtual images 114 a-114 c may be virtual images of internal (e.g.,bones or other organs) or external (e.g., skin or contours thereof)anatomical features. In some implementations, the augmented realitydevice 102 uses three-dimensional data corresponding to the targetindividual 106 to generate the graphical representations 113. In someimplementations, the graphical representations 113 include virtualimages 114 of a detailed underlying anatomy of the face of the targetindividual 106.

A method of displaying the graphical representation 113 may includevarious steps. For example, the method of displaying the graphicalrepresentation 113 may include identifying one or more reference markers112 (FIG. 1B) with the augmented reality device 102 (e.g., 102 a, 102b). The augmented reality device 102 may also determine the distancefrom each reference marker 112 to each of the other reference markers112 and transmit data corresponding to the distances to a processor(e.g., processor 410 of FIG. 4). The augmented reality device 102 mayfurther perform a look up in a database (e.g., database 218 in FIG. 2)with data corresponding to reference markers 112 (e.g., memory 420 ofFIG. 4). The augmented reality device 102 may use the data correspondingto reference markers 112, and in some implementations the plurality oftarget data, to determine data corresponding to the anatomical features'characteristics (e.g., size, location, etc.). For example, the augmentedreality device 102 may use the reference markers 112 and thethree-dimensional visual representation 105 of the target individual 106created from infrared laser scatter beam technology of the image capturedevice 108 to create the graphical representation 113, including thevirtual images 114 a-c of the anatomical features. In particular, theaugmented reality device 102 may transmit the data corresponding to theanatomical features' characteristics to the processor (e.g., processor410 of FIG. 4) and display the graphical representation 113, includingthe virtual images 114 a-c of the anatomical features, on the display110 at a location corresponding to the target individual 106 (see FIG.1C).

As illustrated in FIG. 1C, the second augmented reality device 102 b maydisplay the graphical representation 113 on the eyepiece display 110 b,and the first augmented reality device 102 a may display the graphicalrepresentation 113 on screen 110 a. As previously described, thegraphical representation 113 displayed on screen 110 and overlaid on topof visual representation 105 of the target individual 106 may includecomputer-generated virtual images (e.g., the virtual images 114 a-c ofthe anatomical features). In the example shown, augmented reality device102 displays, on display 110, the graphical representation 113 includesvirtual images 114 a-114 c each representing an organ, bone, orstructure in the human body.

A first virtual image 114 a represents a ribcage of the human body.Virtual image 114 a is overlaid on the visual representation 105 of thetarget individual 106 at a location approximating where the ribcage ofthe target individual 106 is located. A second virtual image 114 brepresents intestines of the human body. Virtual image 114 b is overlaidon the visual representation 105 of the target individual 106 at alocation approximating where the intestines of the target individual 106are located. A third virtual image 114 c represents a pancreas of thehuman body. Virtual image 114 c is overlaid on the visual representation105 of the target individual 106 at a location approximating where thepancreas of the target individual 106 is located. Virtual images 114a-114 c do not represent an exhaustive list of all virtual image, butrather an exemplary list of virtual images that may be displayed by theaugmented reality device 102 in the graphical representation 113.Furthermore, the augmented reality device 102 may display other virtualimages in addition to the virtual images 114 a-114 c, or may omit one ormore virtual images 114 a-114 c, without departing from the teachingsherein.

The graphical representation 113, in combination with the visualrepresentation 105, enhances the view and experience of the user 104 bycreating a composite view of both real and virtual images on the display110. The user 104 may view, through the display 110, the visualrepresentation 105 of the target individual 106 with the virtual images114 a-114 c of organs and other anatomical features represented by thegraphical representation 113 of the target individual 106. Theanatomical features may include organs, bones, muscles, blood vessels,tendons, ligaments, or nerves. In some implementations, the virtualimages 114 a-114 c are not actual images of the internal organs, bones,or other bodily structures of target individual 106, but rather arerepresentative depictions (e.g., illustrations) of those bodilystructures. Allowing the augmented reality device 102 to storerepresentative virtual images 114 a-114 c that can be used for anytarget individual 106 is advantageous because it requires less space(e.g., memory 420) for storing the virtual images 114 a-114 c, allowingthe augmented reality device 102 to operate and display the virtualimages 114 a-114 c at a faster speed.

In some implementations, the augmented reality device 102 may use theimage capture device 108 to take a still picture or record a video ofthe target individual 106. The augmented reality device 102 may thenoverlay virtual images 114 of anatomical features onto the still pictureor recorded video of target individual 106. The augmented reality devicemay then display the still picture or recorded video with the overlaidvirtual images 114 onto the display 110. This may be advantageous,particularly for educational purposes, as it assists a healthcareprofessional in educating a patient about their own anatomy by showingthe patient how their own individual anatomy approximately works. Inother implementations, the augmented reality device 102 may be able totake a picture or record a video of the target individual 106 with thevisual representation 105 that includes the virtual images 114 of organsand other anatomical features. The augmented reality device 102 maystore the picture or recorded video in a storage resource, such as thestorage device 430 (FIG. 4).

With reference now to FIGS. 1D-1F, in some implementations, theaugmented reality device 102 (e.g., 102 a, 102 b) may display theanatomical profile of a target (e.g., target individual 106) based oncertain anticipated changes to the target individual 106. FIG. 1D showsa visual representation 105′ of an alternative target individual 106′displayed on display 110. The augmented reality device 102 may determinethe visual representation 105′ of the alternative target individual 106′with the image capture device 108. As illustrated in FIG. 1E, similar tothe visual representation 105 of FIG. 1B, the augmented reality device102 may identify a plurality of reference markers 112 on the visualrepresentation 105′. In some implementations, the augmented realitydevice 102 may determine an anatomical profile of the alternative targetindividual 106′. The anatomical profile may include a plurality ofcharacteristics corresponding to the alternative target individual 106′.As described above in reference to FIG. 1B, the anatomical profile maybe based on a plurality of target data, input from the user 104, ormachine learning or artificial intelligence algorithms. In someimplementations, the anatomical profile may also be based on certainchanges that the alternative target individual 106′ may undergo. Forexample, the anatomical profile may be based on future weight loss orweight gain that the alternative target individual 106′ will, or desiresto, undergo.

FIGS. 1E and 1F show the augmented reality device 102 displaying thevisual representation 105′ of the alternative target individual 106′ ondisplay 110. The augmented reality device 102 displays the graphicalrepresentation 113 of the alternative target individual 106′, thegraphical representation 113 including virtual images 114 of anatomicalfeatures. In the example shown at FIG. 1F, the augmented reality device102 displays the visual representation 105′ of the alternative targetindividual 106′ if the alternative target individual 106′ had lostweight (FIG. 1F). The augmented reality device also overlays thegraphical representation 113 that includes the virtual images 114 ofanatomical features. The virtual images 114 of anatomical features maybe based on the anatomical features of the alternative target individual106′ if the alternative target individual 106′ lost weight. Furthermore,the visual representation 105′ (FIG. 1F) of the alternative targetindividual 106′ may be based on if the alternative target individual106′ lost weight.

Though FIG. 1F depicts the augmented reality device 102 displaying thevirtual images 114 of anatomical features based on if the alternativetarget individual 106′ lost weight, it should be noted that theaugmented reality device may display virtual images 114 based on othercertain changes, such as gaining weight, becoming pregnant, undergoingreconstructive or cosmetic surgery, or other changes that anindividual's body may undergo. Furthermore, the augmented reality device102 may display other virtual images 114 that show how musculoskeletalfeatures would function under the certain changes.

With reference again to FIGS. 1E and 1F, another aspect of the augmentedreality device 102 is provided. The augmented reality device 102 isconfigured to detect a target individual 106 by using an image capturedevice 108 and display a visual representation 105 of the body of thetarget individual 106. The augmented reality device 102 is configured toidentify one or more reference markers 112 as described above withreference to FIG. 1B. The reference markers 112 are processed toidentify an anatomical profile which includes a plurality of inneranatomical features 114. The augmented reality device 102 is furtherconfigured to access a database 218. The database 218 may be stored inthe augmented reality device 102 or may be stored in a remote server,such as a server located in or accessible through a network 400.

The augmented reality device 102 is configured to display on the display110, a graphical representation of the inner anatomical features ontothe visual representation of the body, as shown in FIG. 1G. Accordingly,a user such as a healthcare professional, is able to identify thelocation of the inner anatomical features without performingconventional procedures such as an x-ray, an MRI or other medicalscanning operations. Such a display is helpful in assisting thehealthcare professional with identifying, within a reasonable degree ofaccuracy, a medical condition, a location for an incision, an organ,specific portion of a muscle or tendon, or any other interior anatomicstructure not externally visible. As an example, augmented realitydevice 102 captures an image of the patient body 106 and displays avisual representation 105 of the body. The augmented reality device 102identifies reference markers 112 a-112 g and searches the database 218to find an anatomical profile 216 corresponding to the spatialorientation and relationship of the identified reference markers 112a-112 g. The graphical representation of the inner anatomical featuresare overlaid onto the visual representation 105 of the body.

The patient may be experiencing abdominal pain and may point to thelocation of the pain on the body. The healthcare professional may thenbe able to better identify the issue by referencing where the pain iswith respect to the location of the inner anatomical features (which isdisplayed in the graphical representation 105), such as distinguishingbetween a pain located at the duodenum as opposed to the pancreas, thepancreatic duct or the like. Accordingly, the augmented reality device102 assists the surgeon with treatment, diagnosis, patient education andthe like without having to subject the patient to an x-ray, MRI or otherinternally invasive scanning procedures.

With reference now to FIG. 1G, another aspect of the augmented realitydevice 102 is provided. The database 218 includes a plurality of futurestate anatomical profiles 216′ corresponding to a selected medicalprocedure 226. It should be appreciated that the database 218 mayinclude thousands of future state anatomical profiles 216′. Theaugmented reality device 102 is further configured to receive theselected medical procedure 226. For instance, the augmented realitydevice 102 may display a menu 118 offering a selection of a plurality ofmedical procedures 216 such as a procedure effecting an organ, bone ormuscle. Such medical procedures include a liposuction, agastrointestinal bypass, a nose job or the like.

In operation, the augmented reality device 102 processes the referencemarkers 112 to determine which anatomical profile 216 stored in thedatabase 218 matches the reference markers 112 identified. The user thenselects a medical procedure 226 and retrieves the future stateanatomical profile 216′ which corresponds to the selected medicalprocedure 225. The augmented reality device 102 then modifies the inneranatomical features 114 based on the selected medical procedure 226. Asan example, FIG. 1B depicts the augmented reality device 102 identifyinga plurality of reference markers 112 on the target individual 106.

With reference now to FIG. 1G the augmented reality device 102 searchesthe database 218 to find an anatomical profile corresponding to thereference markers 112. For example, the augmented reality device 102 mayuse the distance between reference markers 112 to find an anatomicalprofile having similar distances between the reference markers. Forexample, if the patient is a male, that is 5′10″ having the left andright shoulders 112 c and 112 b that are spaced apart from each other 18inches, the left and right hips 112 f and 112 e that are spaced apartfrom each other 19 inches, the augmented reality device 102 searches thedatabase 218 to find an anatomical profile of a male that is 5′10 havingreference markers 112 of similar spacing. It should be appreciated thatthe more reference markers may be used to determine the correspondinganatomical profile other than just the left and right shoulders 112 cand 112 b and the left and right hips 112 f and 112 e. The inneranatomical features are then displayed on the graphical representationas shown in FIG. 1G.

In another aspect, the augmented reality device 102 may be furtherconfigured to scale the inner anatomical features of the selectedanatomical profile based upon the size of the target individual 105relative to the selected anatomical profile. The augmented realitydevice 102 may make a determination that the target individual is largeror smaller than the selected anatomical profile. The augmented realitydevice 102 may be further configured to increase or decrease the size ofthe inner anatomical features associated with the selected anatomicalprofile so as to fit the image and the reference markers 112. As anexample, for a target individual 105 that is larger than the selectedanatomical profile, the inner anatomical features of the selectedanatomical profile is magnified. Such an aspect may be beneficial ininstances where the database 218 is populated with a discrete number ofanatomical profiles. Thus, the augmented reality device 102 isconfigured to display the inner anatomical features onto a visualrepresentation 105 of the target individual 106 using less processingresources relative to an aspect where the database 218 is populated withhundreds of anatomical profiles.

As an example, the database 218 may be populated with only eight (8)anatomical profiles for a respective male and female version of aninfant, child, adult and elderly for a total of sixty-four (64)anatomical profiles. In such an aspect, the augmented reality device 102processes the sixty-four (64) anatomical profiles to select ananatomical profile which matches the spatial dimensions of the referencemarkers 112 the closest. For instance, the target individual 106 may bea male adult that is 5′11″. The database 218 may be populated with amale adult that is 5′8 and another that is 6′2″. The augmented realitydevice 102 selects the anatomical profile of the adult male that is 6′2″in cases where the anatomical profile of the adult male that is 6′2″matches the reference markers 112 more closely than the anatomicalprofile of the adult male that is 5′8″. The augmented reality device 102scales the inner anatomical features of the selected anatomical profileso as to fit within the visual representation 105 of the body of thetarget individual 106. Using the same example, the augmented realitydevice 102 may shrink the inner anatomical features of the selectedanatomical profile so as to fit within the smaller visual representationof the target individual 106.

It should be appreciated that not all of the inner anatomical featuresof the selected anatomical profiles need to be scaled. For instance, theaugmented reality device 102 may be further configured to determinewhich of part of the body of the target individual is larger or smallerthan the selected anatomical profile, wherein the inner anatomicalfeatures of the body part that is larger or smaller than the selectedanatomical profile is scaled. As an example, the augmented realitydevice 102 may magnify the lungs of the selected anatomical profile ininstances where a target individual 106 has a chest cavity that islarger than the selected anatomical profile. The magnification is madeso as to overlay the lung in proportion to visual representation 105 ofthe target individual 106.

In yet another aspect, the augmented reality device 102 may be furtherconfigured to process a body type along with the reference markers 112.For example, it may be assumed that the target individual 106 shown inFIGS. 1A and 1B may be considered a normal body type, the targetindividual 106 shown in FIGS. 1D and 1E may be considered an athleticbody type, and the target individual 106 shown in FIGS. 11 and 1K may beconsidered an overweight body type. The augmented reality device 102processes the body type with a silhouette of the target individual 106and the selected anatomical profile so as to provide an overlay of otheranatomical features such as fatty layer, muscular thickness and thelike. The augmented reality device 102 may be further configured toprocess the silhouette of the target individual and the selectedanatomical profile so as to overlay other features of the inner anatomyof the individual target 106 such as the blood vessels, nerves andorgans corresponding to the selected anatomical profile. It should beappreciated that such a feature provides additional detail to assist thehealthcare professional in providing care, education, perform adiagnosis and the like.

For illustrative purposes, an operation of the augmented reality deviceprocessing a body type and a silhouette along with the selectedanatomical profile is provided. The augmented reality device 102 detectsthe target individual 106 by using an image capture device 108 andprocesses the visual representation 105 of the target individual so asto determine a silhouette. The silhouette may be determined byimplementing any known or later developed image processing technique foredge detection currently known or later developed.

In one aspect, the augmented reality device 102 may process thesilhouette to determine a body type. The body type may be determined byprocessing the spatial relationship between the reference markers, oropposing edges of the silhouette. As an example, by determining that adistance between opposing edges around the waist exceed a distancebetween opposing edges of each shoulder in an image taken from the sameperspective, the reality device 102 may determine that the targetindividual has an overweight body type. As another example, the realitydevice may determine that the target individual 106 has an athletic bodytype by determining that a distance between opposing edges of the waistis smaller than a distance between opposing edges of each shoulder in animage taken from the same perspective. Alternatively, the body type maybe simply inputted by using a drop down menu, a keyboard or any otherinput currently known or later developed.

In cases, where the augmented reality device 102 determines, by imageprocessing or by an input, that the body type is overweight, theaugmented reality device 102 may scale inner anatomical features to becommensurate with the body type. For instance, a fatty layer may be madethicker, relative to a fatty layer of a normal body type, wherein theinner anatomical features of the selected anatomical profiles are scaledto fit behind the thickened fatty layer. The thickening of the fattylayer may be made commensurate with the silhouette, and may be furtherrefined by processing the reference markers 112. In instances where thebody type is athletic, the fatty layer may be made thinner relative to afatty layer of a normal body type, and the muscles enlarged relative toa normal body type. It should be appreciated that the examples providedherein are illustrative and not limiting and that other inner anatomicalfeatures may be adjusted based upon the determined body type.

It should be appreciated that though the description of operation of theaugmented reality device 102 according to the aspect above is providedin a global manner, the reality device 102 may be operated in a granularmanner that is specific to a particular body part. For instance, theadjustment of a fatty layer may be made only to a portion of the bodywhich is deemed to fall outside the specifications of a normal bodytype. For instance, the augmented reality device 102 may determine,based upon the silhouette, that only the waist is overweight and theremaining body is normal. In such an aspect, the augmented realitydevice 102 is configured to only thicken the fatty layer around thewaist and display the fatty layer of the remaining body part as beingnormal. It should be further appreciated that the adjustment of theanatomical features based upon the body type may be done automatically,or based upon an input from the healthcare professional. That is, theaugmented reality device 102 may operate without processing asilhouette. Thus, the graphical presentation of the inner anatomicalfeatures are overlaid onto the visual representation of the body withoutany adjustments made to inner anatomical features such as the fattylayer, muscles, or the like. The healthcare professional, may thenselect an option or otherwise input a desire to modify the graphicalrepresentation with the silhouette of the determined body type.

With reference now to FIG. 1H, the graphical representation 113 ismodified relative to FIG. 1G. In particular, the augmented realitydevice 102 is configured to receive a selected medical procedure 226.For illustrative purposes, the medical procedure 226 is selected by adrop down menu 118 (shown in FIG. 1G). However, other means forinputting the selected medical procedure 226 may be adapted for useherein, illustratively including a keyboard, voice input or the like.For illustrative purposes, the selected medical procedure 226 is agastric bypass. The augmented reality device 102 selects a future stateanatomical profile 216′ corresponding to the selected medical procedure226. In this case, the augmented reality device 102 modifies the inneranatomical feature 114 (namely the stomach) by overlaying the stomachshown in FIG. 1G with the future state of anatomical profile 216′determined by the augmented reality device 102, as shown in FIG. 1H.Accordingly, in this aspect, the patient is able to see the effect asurgical procedure has on the inner anatomical features of the patient.In particular, the patient may be able to see what his inner anatomicalfeatures look like before the medical procedure, as shown in FIG. 1G andafter the medical procedure as shown in FIG. 1H.

In another aspect of the augmented reality device 102, the database 218further includes a plurality of diseases, inherited conditions and/oranatomical variants. As an example, the database may store differenttypes of diseases such as cancer, heart disease, diabetes and the like;inherited conditions such as cystic fibrosis, down syndrome and thelike; anatomical variants may include anatomical variants of anatomicalfeatures such as the gallbladder, kidney, liver and the like. Theaugmented reality device 102 is further configured to determine ananatomical profile corresponding to the identified disease, theidentified inherited condition or the identified anatomical variant andgenerate a graphical representation which a visual representation of thebody modified with the inner anatomical features based on the determinedanatomical profile. As such, the augmented reality device 102 may behelpful in showing the patient and/or surgeon the effects of a diseaseor inherited condition on the inner anatomical features.

In another aspect, the database 218 includes anatomical profiles havingat least one of the diseases, inherited conditions and/or anatomicalvariants. In such an aspect, the augmented reality device 218 simplyselects the anatomical profile corresponding the identified referencemarkers 112 a-112 g having an identified disease, inherited conditionsand/or anatomical variants as the case may be and displays the inneranatomical features associated with the selected anatomical profile. Insuch an embodiment, it is preferable for the patient to identify thedisease, inherited condition or anatomical variant, as the case may be.This may be done by an input field and entered using conventional inputmeans such as a keyboard, voice input, a mouse or the like.

As an example, the surgeon or nurse inputs a disease into the augmentedreality device 218. The augmented reality device 218 detects the targetindividual and displays a visual representation of the body. Theaugmented reality device 218 processes the reference markers 112 a-112 gto identify an anatomical profile having reference markers with the samespatial dimensions and the disease entered into the augmented realitydevice 218. The inner anatomical features of the selected anatomicalprofile is mapped onto the visual representation of the body.

With reference now to FIG. 1I, another aspect of the augmented realitydevice 102 is provided. FIG. 1I depicts the augmented reality device 102mapping a target individual 106. The augmented reality device 102includes an image capture device 108 configured to generate an initialthree-dimensional representation 113′ of the body. For instance, theimage capture device 108 may be a LIDAR camera or an infrared laserscatter beam device. FIG. 1I depicts the initial three-dimensionalrepresentation ‘113 as still images take from the front of the targetindividual 106 and the side of the target individual. However, it shouldbe appreciated that the initial three-dimensional representation 113′depicts the entirety of the target individual 106, and thus may begenerated by taking a number of still images or a video. In one aspect,the target individual 106 may stand on a platform 120 which may rotate360 degrees as indicated by the arrow. The augmented reality device 102is placed a predetermined distance from the target individual 106 andthe image capture device 108 is actuated so as to map the targetindividual. FIG. 1I depicts the images taken as the target individual106 is rotated. It should be appreciated that the initialthree-dimensional representation 113′ of the target individual 106 maybe generated in another manner, for instance the target individual 106may stand still as the augmented reality device 102 rotates about the106. Alternatively, multiple augmented reality devices 102 may bepositioned at different locations around the target individual 106 andthe images captured from each of the augmented reality devices 102 arecompiled together and processed to generate the initialthree-dimensional representation 113′.

The augmented reality device 102 identifies one or more referencemarkers 112 as described above with reference to FIG. 1B. It should beappreciated that the reference markers 112 may be identified in realtime or after the three-dimensional representation of the body has beengenerated. The augmented reality device 102 is further configured todetermine a preferred anatomical profile 216″ based upon the pluralityof reference markers 112. The preferred anatomical profile 216″ being aprofile representative of a fit and healthy human which may be basedupon anatomical features which may be extrapolated from the referencemarkers 112. The anatomical features include not only the referencemarkers 112 but also aspects such as the height and body shape of thetarget individual 106.

With reference now to FIG. 1J, the augmented reality device 102 isfurther configured to modify in three dimensions, the initialthree-dimensional representation 113′ of the body with the preferredanatomical profile 216″ so as to generate a modified initialthree-dimensional representation 228. The modified initialthree-dimensional representation 228 is the initial three-dimensionalrepresentation of the body 113′ which is shaped to conform to the shapeof the preferred anatomical profile 216″. The augmented reality device102 displays a graphical representation of the modified initialthree-dimensional representation 228 of the body.

The augmented reality device 102 may be further configured to receive aselected medical procedure 226 relating to a desired body part, whereinthe desired body part of the initial three-dimensional representation113′ of the body is replaced with a corresponding body part taken fromthe preferred anatomical profile 216″. The augmented reality device 102displays the initial three-dimensional representation 113′ of the bodywith the corresponding desired body part taken from the preferredanatomical profile 216″.

In operation, the augmented reality device 102 processes the referencemarkers 112 to determine which preferred anatomical profile 216″ storedin the database 218 matches the initial three-dimensional representation113′ of the body. The augmented reality device 102 then processes theinitial three-dimensional representation 113′ of the body so as toadjust the contours of the initial three-dimensional representation 113′of the body to match the contours of the preferred anatomical profile216″, which is displayed. As such, the patient can see what they wouldactually look like if their body were adjusted to the preferredanatomical profile. The resulting image is it is not just a preferredprofile overlaid onto the patient, but it is the patient's body shapemodified, stretched, shrunken and the like. Accordingly, it is preferredthat the patient bare as much skin as possible when the initialthree-dimensional representation 113′ of the body is generated, as theskin of the patient will be modified to assume the shape of thepreferred anatomical profile 216″.

With reference again to FIG. 1K, the augmented reality device 102searches the database 218 to find a preferred anatomical profile 216″based on the plurality of reference markers 112. As described above,this may be done by determining the distance between the referencemarkers. It should be appreciated that the database may include hundredsof preferred anatomical profiles 216″ which are representative of a malethat is 5′10″. For instance one of the preferred profiles is of a malethat is 5′10″ having the left and right shoulders 112 c and 112 b thatare spaced apart from each other 18 inches, the left and right hips 112f and 112 e that are spaced apart from each other 19 inches. Another ofthe preferred profiles of a male that is 5′10″ having the left and rightshoulders 112 c and 112 b that are spaced apart from each other 17inches, the left and right hips 112 f and 112 e that are spaced apartfrom each other 17 inches. As stated above, the preferred anatomicalprofiles 216″ are a three-dimensional representation of a fit andhealthy person. The augmented reality device 102 processes the initialthree-dimensional representation 113′ of the body so as to adjust thecontours of the initial three-dimensional representation 113′ of thebody to match the contours of the preferred anatomical profile so as togenerate modified initial three-dimensional representation 228.

FIG. 1J depicts the patient shown in FIG. 1I after the initialthree-dimensional representation 113′ of the body is adjusted to thecontours of the preferred anatomical profile 216″. The modified initialthree-dimensional representation 228 of the body may be displayed on theaugmented reality device 102 and may be casted to another display whichis preferably larger so that the patient can view modified initialthree-dimensional representation 228. The modified initialthree-dimensional representation 228 may be rotated so that the patientcan fully appreciate the change in his or her body. Further, as it isthe initial three-dimensional representation 113′ of the body that isadjusted, the user can appreciate the change in his or her own skin.That is, the initial three-dimensional image 113′ is adjusted, notreplaced by the preferred anatomical profile 216″.

FIG. 1K depicts an embodiment where only a selected body part isadjusted. In one aspect, the augmented reality device 102 may beconfigured to process a selected medical procedure 226. As describedabove, the selected medical procedure 226 may be input by a drop downmenu 118 or a keyboard, voice input or the like. For illustrativepurposes, the selected medical procedure 226 is shown as a liposuctionof the belly fat. Thus, the display 110 depicts the initialthree-dimensional representation 113′ modified with the belly of thepreferred anatomical profile 216″. In another aspect, the user maysimply select a body part to be replaced by the corresponding body partof the preferred anatomical profile 216″ of the body. The user mayselect the body part by touching the body part displayed on a touchscreen display 110, through voice command or other known inputs such asa mouse or a stylus. Accordingly, in this aspect, the patient is able tosee the what his or her actual body would look like should the body bemodified to be the shape and contour of a preferred anatomical profile.In other words, the modified initial three-dimensional representation228 a modification of a selected body part of the initialthree-dimensional representation 113′ with the corresponding body partof the preferred anatomical profile 216″. Accordingly, FIG. 1K depictsthat the arms, neck, legs, etc. of the patient remains the same as shownin FIG. 1I and it is only the belly that is modified to be commensuratewith a belly of the preferred anatomical profile 216″. As such, thepatient will be able to see how he or she looks after a selected medicalprocedure is performed which assists in the patient's decision to haveother procedures performed.

With reference now to FIG. 2, in some implementations, an exemplarysystem 200 for displaying augmented anatomical features (e.g., virtualimages 114 a-114 c) provides the user 104 (e.g., a healthcare provider)with access to an augmented reality module 204 to enhance a user's 104view of a target individual 106. The system 200 may include a network208 that provides access to the augmented reality module 204 thatprovides for the performance of services on remote devices. Accordingly,the network 208 allows for interaction between the user 104 and theaugmented reality module 204. For instance, the augmented reality module204 may provide the user 104 access to the augmented reality module 204and receive event data inputted by the user 104 associated with theuser's 104 interaction with the augmented reality module 204. In turn,augmented reality module 204 may store event data in a storage resource.

The network 208 may include any type of network that allows sending andreceiving communication signals, such as a wireless telecommunicationnetwork, a cellular telephone network, a time division multiple access(TDMA) network, a code division multiple access (CDMA) network, Globalsystem for mobile communications (GSM), a third generation (3G) network,fourth generation (4G) network, a satellite communications network, andother communication networks. The network 208 may include one or more ofa Wide Area Network (WAN), a Local Area Network (LAN), and a PersonalArea Network (PAN). In some examples, the network 208 includes acombination of data networks, telecommunication networks, or acombination of data and telecommunication networks. An augmented realitydevice 102 and augmented reality module 204 communicate with each otherby sending and receiving signals (wired or wireless) via the network208. In some examples, the network 208 provides access to cloudcomputing resources, which may be elastic/on-demand computing and/orstorage resources available over the network 208. The term ‘cloud’services generally refers to a service performed not locally on a user'sdevice (e.g., device 102), but rather delivered from one or more remotedevices accessible via one or more networks 208.

The augmented reality device 102 may include, but is not limited to, aportable electronic device (e.g., smartphone, cellular phone, personaldigital assistant, personal computer, or wireless tablet device), awearable augmented reality device, or any other electronic devicecapable of capturing images and overlaying computer-generated or virtualimages (e.g., virtual images 114 a-114 c) over a real world view (e.g.,the visual representation 105). The augmented reality device 102includes data processing hardware (a computing device that executesinstructions), memory hardware, and a display 110 in communication withthe data processing hardware. Input module 214 provides the user 104access to interacting with the augmented reality module 204 through theaugmented reality device 102. In some examples, the input module 214includes a keyboard, touchpad, mouse, microphones, eye-tracking device,gesture tracking device, and/or a camera for allowing the user 104 toinput data. In addition to or in lieu of the display 110, augmentedreality device 102 may include one or more speakers to output audio datato the user 104.

In some implementations, the user 104 may interact with the input module214 by inputting data corresponding to reference markers 224. Thereference markers 224 may correspond to locations on the targetindividual 106. Data corresponding to the reference markers 224 is thensent to the augmented reality module 204. The augmented reality module204 may communicate with an anatomy module 216. For instance, theaugmented reality module 204 may send anatomical data 222 correspondingto the reference markers 224 to the anatomy module 216. The augmentedreality module 204 may then request data corresponding to visualrepresentations (e.g., virtual images 114 a-114 c) of anatomicalfeatures from the anatomy module 216. The anatomy module 216 may thenretrieve data corresponding to the visual representations of anatomicalfeatures, future state anatomical profile or preferred anatomicalprofile from the database 218. The anatomy module 216 may then generatean anatomical profile 220 to be displayed on the display 110.

The augmented reality device 102 may include the image capture device108 having a detection module 210 and a tracking module 212. Thedetection module 210 and tracking module 212 may obtain visual datacorresponding to the target individual 106 and send it to the augmentedreality module 204. The visual data may be data corresponding to thecurrent real world view of the image capture device 108 and may includedata such as the distance between the target individual 106 and theaugmented reality device 102, whether the target individual 106 is inmotion or at rest, or any other data that corresponds to thevisualization of the target individual 106 by the augmented realitydevice 102. The image capture device 108 may send data from thedetection module 210 and the tracking module 212 to the augmentedreality module 204. The augmented reality module 204 may use the datafrom the detection module 210 and the tracking module 212 correspondingto the image capture device's 108 real world view, coupled with virtualimages of the anatomical profile 216 to create a composite enhanced viewof the target individual 106 on the display 110.

FIG. 3A is a flow chart illustrating a method 300 for displayingaugmented anatomical features (e.g., virtual images 114 a-114 c) inaccordance with an example implementation of the disclosed technology.According to one example, the method 300 may be performed by anelectronic device, such as the augmented reality device 102. The method300 begins at block 302 where the augmented reality device detects atarget individual (e.g., target individual 106) and tracks the targetindividual. The target individual has a body. For example, at block 302,the image capture device 108 a, 108 b may detect and/or track the targetindividual 106 via the detection module 210 and/or the tracking module212.

At block 304, the method includes displaying, on a display (e.g.,display 110), a visual representation of the body. For example, at block302, the display 110 a, 110 b may receive and display the visualrepresentation 105, 105′ from the augmented reality module 204. In someimplementations, after block 304, the method may return to block 302. Inother implementations, the method may advance to block 306.

At block 306, the method includes identifying a plurality of referencemarkers (e.g., reference markers 112) on the visual representation(e.g., the visual representation 105, 105′) of the body. For example,the augmented reality device 102 may receive the reference markers 112from the user's interaction with the display 110 a, 110 b, machinelearning, or another previously-described method. In someimplementations, the augmented reality device 102 may receive an inputof reference markers 112 from the user 104 and receive and generate thegraphical representation 113 from the augmented reality module 204.

At block 308, the method includes determining, at a processor, ananatomical profile (e.g., anatomical profile 216) of the targetindividual based on the plurality of reference markers. The anatomicalprofile may include a plurality of characteristics (e.g., age, gender,etc.) corresponding to the target individual. In some implementations,the augmented reality device 102 receives an anatomical profile based onthe reference markers 112. For example, the anatomy module 216 maytransmit the anatomical profile to the augmented reality device 102through the augmented reality module 204.

At block 310, the method includes displaying, on the display, graphicalrepresentations (e.g., virtual images 114 a-114 c) of the anatomicalfeatures overlaid on the visual representation 105, 105′ of the body.The graphical representations of the anatomical features may be orientedon the visual representation of the body based on the anatomicalprofile. In some implementations, the augmented reality device generatesand displays a graphical representation of the anatomical profileoverlaid on the visual representation of the target individual. In oneaspect, the anatomical profile includes inner anatomical features, andthe graphical representation of the inner anatomical features of theanatomical profile are overlaid on the visual representation of thetarget individual. Following block 310, the method 300 may conclude.

FIG. 3B is a flow chart illustrating a method 300B for displaying amodified inner anatomical features of a patient based on a selectedmedical procedure. According to one example, the method 300B may beperformed by an electronic device, such as the augmented reality device102. The method 300B begins at block 302B where the augmented realitydevice detects a target individual (e.g., target individual 106). Thetarget individual has a body. For example, at block 302, the imagecapture device 108 a, 108 b may detect the target individual 106 via thedetection module 210 and/or the tracking module 212.

At block 304B, the method includes displaying, on a display (e.g.,display 110), a visual representation of the body. For example, at block302, the display 110 a, 110 b may receive and display the visualrepresentation 105, 105′ from the augmented reality module 204. In someimplementations, after block 304, the method may return to block 302. Inother implementations, the method may advance to block 306B.

At block 306B, the method includes identifying a plurality of referencemarkers (e.g., reference markers 112) on the visual representation(e.g., the visual representation 105, 105′) of the body. For example,the augmented reality device 102 may receive the reference markers 112from the user's interaction with the display 110 a, 110 b, machinelearning, or another previously-described method. In someimplementations, the augmented reality device 102 may receive an inputof reference markers 112 from the user 104 and receive and generate thegraphical representation 113 from the augmented reality module 204.

At block 308B, the method includes determining, at a processor, ananatomical profile (e.g., anatomical profile 216) of the targetindividual based on the plurality of reference markers, the anatomicalprofile includes a plurality of inner anatomical features. Theanatomical profile may include a plurality of characteristics (e.g.,age, gender, etc.) corresponding to the target individual. In someimplementations, the augmented reality device 102 receives an anatomicalprofile based on the reference markers 112. For example, the anatomymodule 216 may transmit the anatomical profile to the augmented realitydevice 102 through the augmented reality module 204.

At block 310B, the method includes selecting a medical procedure. Themedical procedure may be selected by choosing from a drop down menu, ormay be verbally inputted into an augmented reality device 102 using akeyboard, voice command or the like. The medical procedure is aprocedure effecting an organ, bone or muscle. Such medical proceduresinclude a liposuction, a gastrointestinal bypass, a nose job or thelike.

At block 312B, the method includes the step of determining a futurestate anatomical profile corresponding to the selected medicalprocedure. For instance, the future state anatomical profile may be oneof thousands of future state anatomical profiles stored in the database218 which corresponds to the anatomical profile.

At block 314B, the method includes displaying, on the display, graphicalrepresentations (e.g., virtual images 114 a-114 c) of the visualrepresentation of the body modified with the inner anatomical featuresbased on the selected medical procedure.

FIG. 3C is a flow chart illustrating a method 300C for displaying avisual three-dimensional depiction of a patient having a body of apreferred anatomical profile. According to one example, the method 300Cmay be performed by an electronic device, such as the augmented realitydevice 102. The method 300C begins at block 302C where the augmentedreality device maps a target individual (e.g., target individual 106) soas to generate an initial three-dimensional representation of the body.The target individual has a body. For example, at block 302, the imagecapture device 108 a, 108 b may detect the target individual 106 via thedetection module 210 and/or the tracking module 212.

At block 304C, the method includes identifying a plurality of referencemarkers (e.g., reference markers 112) on the visual representation(e.g., the visual representation 105, 105′) of the body. For example,the augmented reality device 102 may receive the reference markers 112from the user's interaction with the display 110 a, 110 b, machinelearning, or another previously-described method. In someimplementations, the augmented reality device 102 may receive an inputof reference markers 112 from the user 104 and receive and generate thegraphical representation 113 from the augmented reality module 204.

At block 306C, the method includes determining, at a processor, apreferred anatomical profile (e.g., anatomical profile 216) of thetarget individual based on the plurality of reference markers, thepreferred anatomical profile being a preferred three-dimensionalrepresentation of the body that is a body of a fit or healthy person.The anatomical profile may include a plurality of characteristics (e.g.,age, gender, etc.) corresponding to the target individual. In someimplementations, the augmented reality device 102 receives a preferredanatomical profile based on the reference markers 112. For example, theanatomy module 216 may transmit the preferred anatomical profile to theaugmented reality device 102 through the augmented reality module 204.

At block 308C, the method includes modifying, in three dimensions, theinitial three-dimensional representation 113′ of the body so as to havea shape of the preferred anatomical profile. At block 310C, the methodincludes the step of displaying the modified initial three-dimensionalrepresentation of the body. As such, the patient can see what they wouldactually look like if their body were adjusted to the preferredanatomical profile. The resulting image is it is not just a preferredprofile overlaid onto the patient, but it is the patient's body shapethat is modified.

FIG. 4 is schematic view of an example computing device 400 that may beused to implement the systems and methods described in this document.The computing device 400 is intended to represent various forms ofdigital computers, such as laptops, desktops, workstations, personaldigital assistants, servers, blade servers, mainframes, and otherappropriate computers. The components shown here, their connections andrelationships, and their functions, are meant to be exemplary only, andare not meant to limit implementations of the inventions describedand/or claimed in this document.

The computing device 400 includes a processor 410, memory 420, a storagedevice 430, a high-speed interface/controller 440 connecting to thememory 420 and high-speed expansion ports 450, and a low speedinterface/controller 460 connecting to a low speed bus 470 and a storagedevice 430. Each of the components 410, 420, 430, 440, 450, and 460, areinterconnected using various busses, and may be mounted on a commonmotherboard or in other manners as appropriate. The processor 410 canprocess instructions for execution within the computing device 400,including instructions stored in the memory 420 or on the storage device430 to display graphical information for a graphical user interface(GUI) on an external input/output device, such as display 480 coupled tohigh speed interface 440. In other implementations, multiple processorsand/or multiple buses may be used, as appropriate, along with multiplememories and types of memory. Also, multiple computing devices 400 maybe connected, with each device providing portions of the necessaryoperations (e.g., as a server bank, a group of blade servers, or amulti-processor system).

The memory 420 stores information non-transitorily within the computingdevice 400. The memory 420 may be a computer-readable medium, a volatilememory unit(s), or non-volatile memory unit(s). The non-transitorymemory 420 may be physical devices used to store programs (e.g.,sequences of instructions) or data (e.g., program state information) ona temporary or permanent basis for use by the computing device 400.Examples of non-volatile memory include, but are not limited to, flashmemory and read-only memory (ROM)/programmable read-only memory(PROM)/erasable programmable read-only memory (EPROM)/electronicallyerasable programmable read-only memory (EEPROM) (e.g., typically usedfor firmware, such as boot programs). Examples of volatile memoryinclude, but are not limited to, random access memory (RAM), dynamicrandom access memory (DRAM), static random access memory (SRAM), phasechange memory (PCM) as well as disks or tapes.

The storage device 430 is capable of providing mass storage for thecomputing device 400. In some implementations, the storage device 430 isa computer-readable medium. In various different implementations, thestorage device 430 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device, a flash memory or other similarsolid state memory device, or an array of devices, including devices ina storage area network or other configurations. In additionalimplementations, a computer program product is tangibly embodied in aninformation carrier. The computer program product contains instructionsthat, when executed, perform one or more methods, such as thosedescribed above. The information carrier is a computer- ormachine-readable medium, such as the memory 420, the storage device 430,or memory on processor 410.

The high speed controller 440 manages bandwidth-intensive operations forthe computing device 400, while the low speed controller 460 manageslower bandwidth-intensive operations. Such allocation of duties isexemplary only. In some implementations, the high-speed controller 440is coupled to the memory 420, the display 480 (e.g., through a graphicsprocessor or accelerator), and to the high-speed expansion ports 450,which may accept various expansion cards (not shown). In someimplementations, the low-speed controller 460 is coupled to the storagedevice 430 and a low-speed expansion port 490. The low-speed expansionport 490, which may include various communication ports (e.g., USB,Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or moreinput/output devices, such as a keyboard, a pointing device, a scanner,or a networking device such as a switch or router, e.g., through anetwork adapter.

The computing device 400 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 400 a or multiple times in a group of such servers 400a, as a laptop computer 400 b, or as part of a rack server system 400 c.

Among other advantages, the present disclosure provides methods, userdevices, and systems for displaying augmented anatomical features. Anaugmented reality device may overlay virtual images of anatomy on top ofthe human body illustrate an approximation of the structures, tissues ororgans that lie beneath the surface of an individual, such as a targetindividual, in front of a user, such as a healthcare professional. Thevirtual images can be adjusted to fit the target individual. The usermay use the augmented reality device to identify certain anatomicalreference points on the body of the target individual, and use thosepoints to anchor and adjust the virtual images over the targetindividual. The virtual images may be representative of human anatomy ofa human of similar age, sex, etc.

Among other advantages, the present disclosure also provides a method,user device, and system that does not require input of data files fromoutside imaging (e.g., x-ray, magnetic resonance imaging, computedtomography scan, etc.). Such files may have incompatible formats, belarge and unwieldy, or require a large amount of processing power foreach target individual.

Among other advantages, the present disclosure also provides a method,user device, and system that may be for general use. In this regard, useof the augmented reality device may not be restricted to certifiedhealthcare providers. Furthermore, the expectation of the augmentedreality device may be to output or display a computer-generatedapproximation of a representative human anatomy.

Among other advantages, the present disclosure also provides broadapplicability. The augmented reality device may be in constant and rapiduse with one target individual after another, and without requiring theinput of outside data.

Various implementations of the systems and techniques described hereincan be realized in digital electronic and/or optical circuitry,integrated circuitry, specially designed ASICs (application specificintegrated circuits), computer hardware, firmware, software, and/orcombinations thereof. These various implementations can includeimplementation in one or more computer programs that are executableand/or interpretable on a programmable system including at least oneprogrammable processor, which may be special or general purpose, coupledto receive data and instructions from, and to transmit data andinstructions to, a storage system, at least one input device, and atleast one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium” and“computer-readable medium” refer to any computer program product,non-transitory computer readable medium, apparatus and/or device (e.g.,magnetic discs, optical disks, memory, Programmable Logic Devices(PLDs)) used to provide machine instructions and/or data to aprogrammable processor, including a machine-readable medium thatreceives machine instructions as a machine-readable signal. The term“machine-readable signal” refers to any signal used to provide machineinstructions and/or data to a programmable processor.

The processes and logic flows described in this specification can beperformed by one or more programmable processors, also referred to asdata processing hardware, executing one or more computer programs toperform functions by operating on input data and generating output. Theprocesses and logic flows can also be performed by special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit). Processors suitable for theexecution of a computer program include, by way of example, both generaland special purpose microprocessors, and any one or more processors ofany kind of digital computer. Generally, a processor will receiveinstructions and data from a read only memory or a random access memoryor both. The essential elements of a computer are a processor forperforming instructions and one or more memory devices for storinginstructions and data. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto optical disks, or optical disks. However, a computer need nothave such devices. Computer readable media suitable for storing computerprogram instructions and data include all forms of non-volatile memory,media and memory devices, including by way of example semiconductormemory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto opticaldisks; and CD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, one or more aspects of thedisclosure can be implemented on a computer having a display device,e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, ortouch screen for displaying information to the user and optionally akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

A software application (i.e., a software resource) may refer to computersoftware that causes a computing device to perform a task. In someexamples, a software application may be referred to as an “application,”an “app,” or a “program.” Example applications include, but are notlimited to, system diagnostic applications, system managementapplications, system maintenance applications, word processingapplications, spreadsheet applications, messaging applications, mediastreaming applications, social networking applications, and gamingapplications.

The non-transitory memory may be physical devices used to store programs(e.g., sequences of instructions) or data (e.g., program stateinformation) on a temporary or permanent basis for use by a computingdevice. The non-transitory memory may be volatile and/or non-volatileaddressable semiconductor memory. Examples of non-volatile memoryinclude, but are not limited to, flash memory and read-only memory(ROM)/programmable read-only memory (PROM)/erasable programmableread-only memory (EPROM)/electronically erasable programmable read-onlymemory (EEPROM) (e.g., typically used for firmware, such as bootprograms). Examples of volatile memory include, but are not limited to,random access memory (RAM), dynamic random access memory (DRAM), staticrandom access memory (SRAM), phase change memory (PCM) as well as disksor tapes.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. Accordingly, otherimplementations are within the scope of the following claims.

What is claimed is:
 1. A method comprising: detecting a targetindividual having a body; displaying, on a display, a visualrepresentation of the body; identifying a plurality of reference markerson the visual representation of the body; determining, at a processor,an anatomical profile of the target individual based on the plurality ofreference markers, the anatomical profile including a plurality of inneranatomical features; and displaying, on the display, a graphicalrepresentation of the inner anatomical features onto the visualrepresentation of the body so as to assist in the identification of theinner anatomical features.
 2. The method of claim 1, wherein theplurality of reference markers correspond to at least one of a navel, aportion of a sternum, a portion of a hip, a portion of a collarbone, aportion of a shoulder, lips, corners of the mouth, tip of the nose, orears.
 3. The method of claim 1, wherein the anatomical profile of thetarget individual is further based on a plurality of data correspondingto the body.
 4. The method of claim 1, wherein the plurality of inneranatomical features includes at least one of organs, bones or muscles.5. The method of claim 1, wherein the plurality of reference markers areidentified by a user interacting with the display.
 6. The method ofclaim 1, further including the step of selecting a medical procedure anddetermining a future state anatomical profile corresponding to theselected medical procedure, wherein the graphical representationincludes a visual representation of the body modified with the inneranatomical features based on the selected medical procedure.
 7. Themethod of claim 6, in some aspect, the method further includes providinga graphical representation of the body modified with outer anatomicalfeatures based on the selected medical procedure.
 8. The method of claim6, wherein the selected medical procedure is a procedure effecting atleast one of organs, bones or muscles.
 8. The method of claim 6, whereinthe plurality of reference markers correspond to at least one of anavel, a portion of a sternum, a portion of a hip, a portion of acollarbone, a portion of a shoulder, lips, corners of the mouth, tip ofthe nose, or ears.
 9. The method of claim 6, wherein the anatomicalprofile of the target individual is further based on a plurality of datacorresponding to the body.
 10. The method of claim 6, wherein theplurality of inner anatomical features includes at least one of organs,bones or muscles.
 11. The method of claim 6, wherein the plurality ofreference markers are identified by a user interacting with the display.12. The method of claim 1, further including the step of identifying atleast one of a disease, an inherited condition or an anatomical variant,wherein the anatomical profile includes the disease, inherited conditionor anatomical variant, and determining the anatomical profilecorresponding to the identified disease, the identified inheritedcondition or the identified anatomical variant and the identifiedplurality of reference markers, wherein the graphical representationincludes a visual representation of the body modified with the inneranatomical features based on determined anatomical profile.
 13. Themethod of claim 12, wherein the disease, the inherited condition and theanatomical variant effects at least one of organs, bones or muscles. 14.The method of claim 12, wherein the plurality of reference markerscorrespond to at least one of a navel, a portion of a sternum, a portionof a hip, a portion of a collarbone, a portion of a shoulder, lips,corners of the mouth, tip of the nose, or ears.
 15. The method of claim12, wherein the anatomical profile of the target individual is furtherbased on a plurality of data corresponding to the body.
 16. The methodof claim 12, wherein the plurality of inner anatomical features includesat least one of organs, bones or muscles.
 17. The method of claim 12,wherein the plurality of reference markers are identified by a userinteracting with the display.
 18. The method of claim 1, furtherincluding, scaling the inner anatomical features to fit the visualrepresentation of the body.
 19. A user device comprising: a display;data processing hardware in communication with the display; and memoryhardware in communication with the data processing hardware, the memoryhardware storing instructions that when executed on the data processinghardware cause the data processing hardware to perform operationsincluding: detecting a target individual having a body; displaying, onthe display, a visual representation of the body; identifying aplurality of reference markers on the visual representation of the body;determining an anatomical profile of the target individual based on theplurality of reference markers, the anatomical profile including aplurality of inner anatomical features; and displaying, on the display,a graphical representation of the inner anatomical features onto thevisual representation of the body so as to assist in the identificationof the inner anatomical features.
 20. The user device of claim 19,wherein the plurality of reference markers correspond to at least one ofa navel, a portion of a sternum, a portion of a hip, a portion of acollarbone, or a portion of a shoulder.
 21. The user device of claim 19,wherein the anatomical profile of the target individual is further basedon a plurality of data corresponding to the body.
 22. The user device ofclaim 19, wherein the plurality of inner anatomical features includes atleast one of organs, bones or muscles.
 23. The user device of claim 19,wherein the plurality of reference markers are identified by a userinteracting with the display.
 24. The user device of claim 19, whereinthe plurality of reference markers are identified by the data processinghardware.
 25. The user device of claim 19, wherein the operationsfurther include selecting a medical procedure and determining a futurestate anatomical profile corresponding to the selected medicalprocedure, wherein the graphical representation includes a visualrepresentation of the body modified with the inner anatomical featuresbased on the selected medical procedure.
 26. The user device of claim25, wherein the selected medical procedure is a procedure at least oneof organs, bones or muscles.
 27. The user device of claim 19, whereinthe operations further identifying at least one of a disease, aninherited condition or an anatomical variant, wherein the anatomicalprofile includes the disease, inherited condition or anatomical variant,and determining the anatomical profile corresponding to the identifieddisease, the identified inherited condition or the identified anatomicalvariant and the identified plurality of reference markers, wherein thegraphical representation includes a visual representation of the bodymodified with the inner anatomical features based on determinedanatomical profile.
 28. The user device as set forth in claim 27,wherein the disease, the inherited condition and the anatomical varianteffects at least one of organs, bones or muscles.
 29. The user device asset forth in claim 19, wherein the operations further include scalingthe inner anatomical features to fit the visual representation of thebody.
 30. A system comprising: a user device including: a first display;data processing hardware in communication with the first display; memoryhardware in communication with the data processing hardware, the memoryhardware storing instructions that when executed on the data processinghardware cause the data processing hardware to perform operationsincluding: detecting a target individual having a body; displaying, on adisplay, a visual representation of the body; identifying a plurality ofreference markers on the visual representation of the body; determining,at a processor, an anatomical profile of the target individual based onthe plurality of reference markers, the anatomical profile including aplurality of inner anatomical features; and a wearable in communicationwith the user device, the wearable including a second display configuredto display a graphical representation of the inner anatomical featuresonto the visual representation of the body so as to assist in theidentification of the inner anatomical features.
 31. The system of claim30, wherein the plurality of reference markers correspond to at leastone of a navel, a portion of a sternum, a portion of a hip, a portion ofa collarbone, or a portion of a shoulder.
 32. The system of claim 30,wherein the anatomical profile of the target individual is further basedon a plurality of data corresponding to the body.
 33. The system ofclaim 30, wherein the plurality of inner anatomical features includes atleast one of organs, bones or muscles.
 34. The system of claim 30,wherein the plurality of reference markers are identified by a userinteracting with the display.
 35. The system of claim 30, wherein thewearable is further configured to detect and map the target individualindependently of the user device.
 36. The system of claim 30, whereinthe selected medical procedure is a procedure effecting an organ. 37.The system of claim 30, wherein the operations further include selectinga medical procedure and determining a future state anatomical profilecorresponding to the selected medical procedure, wherein the graphicalrepresentation includes a visual representation of the body modifiedwith the inner anatomical features based on the selected medicalprocedure.
 38. The system of claim 30, wherein the operations furtheridentifying at least one of a disease, an inherited condition or ananatomical variant, wherein the anatomical profile includes the disease,inherited condition or anatomical variant, and determining theanatomical profile corresponding to the identified disease, theidentified inherited condition or the identified anatomical variant andthe identified plurality of reference markers, wherein the graphicalrepresentation includes a visual representation of the body modifiedwith the inner anatomical features based on determined anatomicalprofile.
 40. The system of claim 30, wherein the operations furtherinclude scaling the inner anatomical features to fit the visualrepresentation of the body.
 41. A method comprising the steps of:mapping a target individual having a body; generating an initialthree-dimensional representation of the body based on the mapping;identifying a plurality of reference markers on the initialthree-dimensional representation of the body; determining, at aprocessor, a preferred anatomical profile of the target individual basedon the plurality of reference markers, the preferred anatomical profilebeing a preferred three-dimensional representation of the body;modifying in three dimensions, at the processor, the initialthree-dimensional representation of the body so as to have a shape ofthe preferred anatomical profile; and displaying, on the display, themodified initial three-dimensional representation of the body.
 42. Themethod as set forth in claim 41, wherein the plurality of referencemarkers correspond to at least one of a navel, a portion of a sternum, aportion of a hip, a portion of a collarbone, a portion of a shoulder,lips, corners of the mouth, tip of the nose, or ears.
 43. The method asset forth in claim 41, further including the step of selecting a medicalprocedure relating to a desired body part; and replacing the desiredbody part with a corresponding desired body part taken from thepreferred anatomical profile, and displaying the initialthree-dimensional representation of the body with the correspondingdesired body part.
 44. The method as set forth in claim 43, wherein themedical procedure is one selected from the list consisting of aliposuction, a breast enhancement, and a tummy tuck.
 45. The method asset forth in claim 44, wherein the future anatomical image is displayedin three-dimensions.
 46. A user device comprising: a display; dataprocessing hardware in communication with the display; and memoryhardware in communication with the data processing hardware, the memoryhardware storing instructions that when executed on the data processinghardware cause the data processing hardware to perform operationsincluding: mapping a target individual having a body; generating aninitial three-dimensional representation of the body based on themapping, the initial three-dimensional representation; identifying aplurality of reference markers on the initial three-dimensionalrepresentation of the body; determining, at a processor, a preferredanatomical profile of the target individual based on the plurality ofreference markers, the preferred anatomical profile being a preferredthree-dimensional representation of the body; modifying in threedimensions, at the processor, the initial three-dimensionalrepresentation of the body so as to have a shape of the preferredanatomical profile; and displaying, on the display, the modified initialthree-dimensional representation of the body.
 47. The user device as setforth in claim 46, wherein the plurality of reference markers correspondto at least one of a navel, a portion of a sternum, a portion of a hip,a portion of a collarbone, or a portion of a shoulder.
 48. The userdevice as set forth in claim 46, further including the step of selectinga medical procedure relating to a desired body part; and replacing thedesired body part with a corresponding desired body part taken from thepreferred anatomical profile so as to generate a future anatomical imageand displaying the future anatomical image.
 49. The user device as setforth in claim 48, wherein the medical procedure is one selected fromthe list consisting of a liposuction, a breast enhancement, and a tummytuck.
 50. The user device as set forth in claim 49, wherein the futureanatomical image is displayed in three-dimensions.
 51. A systemcomprising: a user device including: a first display; data processinghardware in communication with the display; and memory hardware incommunication with the data processing hardware, the memory hardwarestoring instructions that when executed on the data processing hardwarecause the data processing hardware to perform operations including:mapping a target individual having a body; generating an initialthree-dimensional representation of the body based on the mapping, theinitial three-dimensional representation; identifying a plurality ofreference markers on the initial three-dimensional representation of thebody; determining, at a processor, a preferred anatomical profile of thetarget individual based on the plurality of reference markers, thepreferred anatomical profile being a preferred three-dimensionalrepresentation of the body; modifying in three dimensions, at theprocessor, the initial three-dimensional representation of the body soas to have a shape of the preferred anatomical profile; and a wearablein communication with the user device, the wearable including a seconddisplay configured to display, on the display, the modified initialthree-dimensional representation of the body.
 52. The system as setforth in claim 51, wherein the plurality of reference markers correspondto at least one of a navel, a portion of a sternum, a portion of a hip,a portion of a collarbone, a portion of a shoulder, lips, corners of themouth, tip of the nose, or ears.
 53. The system as set forth in claim51, further including the step of selecting a medical procedure relatingto a desired body part; and replacing the desired body part with acorresponding desired body part taken from the preferred anatomicalprofile so as to generate a future anatomical image and displaying thefuture anatomical image.
 54. The system as set forth in claim 53,wherein the medical procedure is one selected from the list consistingof a liposuction, a breast enhancement, and a tummy tuck.
 55. The systemas set forth in claim 54, wherein the future anatomical image isdisplayed in three-dimensions.